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Why Theories of Change Matter 

Mark R. Connolly and Elaine Seymour 

What is a Theory of Change? 

Mindfully or not, people are theorists of change. That is, they are theorists insofar that they 
engage in a mental process by which they develop ideas that allow them to explain why events 
ought to occur (Turner, 1982). As a way of managing the uncertainty of everyday living, people 
rely on personal theories, or predictive assumptions, about the best ways to achieve desired 
effects. Personal theories about what kinds of action will bring about desired changes and why 
some actions work best are but some of many forms of a person’s tacit knowledge and thus 
typically remain unstated. For the purposes of this paper, we work from this definition: A theory 
of change is a predictive assumption about the relationship between desired changes and the 
actions that may produce those changes. Putting it another way, “If I do x, then I expect y to 
occur, and for these reasons .” 

Theories of change can drive programs as well as people. A program seeking to effect change 
or reform often tacitly reflects the theories of change of the program’s designers. Because 
reformers tend to jump from identifying a problem to choosing ways of ameliorating it, they 
often do not articulate the reasons why those strategies will achieve the desired changes—that is, 
the program’s theory of change. Theories of change matter because they are usually implicit, and 
what remains unseen cannot be questioned. 

A crucial factor in designing successful reform efforts is making programmatic theories of 
change explicit. Evaluators and grant-making organizations, which are especially interested in 
why changes do or do not occur as hoped, have found that a powerful way to improve the 
chances that a set of activities or program of action will succeed is to help the organizers specify 
the reasoning that serves as their theory of change (Connell & Kubisch, 1998; Sullivan & 

Stewart, 2008; Weiss, 1995). Doing this can expose predictive assumptions that do not hold up 
for various reasons. Among the most common pitfalls are not basing implied or stated theories of 
change in reality or evidence, failing to consider plausible alternate explanations, relying on 
limited perspectives, and basing them exclusively on strong affective commitments. 

To demonstrate the value of explicating programmatic theories of change, evaluation 
researcher Carol Weiss (1995) uses as an example a job-training program for disadvantaged 
youth. To help these youths avoid negative social experiences (e.g., criminal activity, illicit drug 
use), the program sought to teach them “job readiness skills,” such as dressing appropriately, 
being punctual, and getting along with supervisors and coworkers. The logic on which this 
program was based consisted of a chain of assumptions, of which these are a few: 

• Training for attractive occupations is (or can be) provided in accessible locations. 

• Information about the program’s availability will reach the target audience. 
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• When young people hear of the program’s availability, they will sign up for it and attend 
regularly. 

• Trainers will offer quality training and help youth leam marketable skills. 

• Youth will internalize the values and absorb the knowledge. 

• Having attained the knowledge and skills, the youth will seek a job. 

• Jobs with adequate pay will be available to the participants. 

• Employers will hire the participants to fill the jobs. 

• The youth will perform well, and employers will be supportive. 

• Youth will become regular workers and wage earners. 

• Youth will not engage in socially undesirable behaviors such as drug use, crime, and so 
forth. 

As Weiss (1995) points out, making this reasoning explicit shows which assumptions may be 
problematic. For example, people with experience with running this kind of program will point 
out that instruction is often subpar and that finding dependable trainers is difficult. Moreover, 
depending on the setting and economic circumstances, it may be the case that few job 
opportunities are available even for those program participants who indeed leam positive job 
readiness skills. 

By making explicit the assumptions that constitute a program’s theory of change, it becomes 
possible to improve the “bet” made by program designers and funders that taking particular 
courses of action will achieve the outcomes they desire. Therefore, one way to increase the 
chances that a change initiative—such as reforming undergraduate STEM education—will 
succeed is to explicate its theory of change and then critically examine its reasoning about causes 
and effects. 

Analyzing Theories of Change in a Sample of STEM Education Reform Programs 

To illustrate the way in which STEM reform efforts are guided by theories of change that are 
often implicit, we followed Weiss’s example and extracted the theories of change embedded in 
nine projects funded by the National Science Foundation (NSF). We focused on three programs 
that sought to improve STEM education: the Math-Science Partnerships (MSP), Course, 
Curriculum, and Laboratory Improvement (CCLI), and the Systemic Changes in Undergraduate 
Chemistry Curriculum initiative. From each, we selected three projects that varied by discipline, 
target of change, geographical region, and so forth. Using program solicitations and project 
summaries that are publicly available through the NSF website, we inferred the theory of change 
within each program and in funded projects that we sampled from each program. We describe 
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each program and project sufficiently to clarify the theories of change that inform particular 
strategies of action. 

I. Math and Science Partnership Program 

Since 2002, the MSP program has sought to improve learning outcomes in mathematics and 
science by all students, at all preK-12 levels. Claiming that there is a “close relationship between 
student achievement and teacher knowledge and teaching skills” (p. 1), the authors of the MSP 
program solicitation argued that providing excellent education in math, science, and technology 
depends significantly on the quality of the preK-12 instructional workforce—namely, well- 
prepared and well-supported schoolteachers. Because high quality teacher preparation and 
professional development are necessary but not sufficient for improving student performance in 
math and science, systemic change of math and science education must address “other essential 
components of the educational system [that] include the availability of a challenging curriculum 
and instructional materials, the judicious use of technology to support instruction, and 
assessment systems ... that inform classroom instruction” (p. 2). Believing that “student learning 
also depends on successful interactions among leadership, resources/partnerships, 
policy/infrastructure, strategic decisions/interventions, [program] sustainability, and 
outcomes/evaluation” (p. 2), the MSP sought to foster partnerships between school districts and 
institutions of higher education as well as other stakeholders (e.g., community organizations, 
private foundations, professional societies, education research organizations, and so forth). The 
insistence that higher education must play a critical role in preK-12 education reform 
distinguishes the MSP program from other NSF-supported systemic education reform efforts. 

Drawing on these assumptions about how elements of the educational system are related, the 
2002 program solicitation claimed that the MSP program would achieve its long-tenn outcomes 
by supporting exemplary partnerships that address these four goals: 

{Goal 1 ) To significantly enhance the capacity of schools to provide a challenging 
curriculum for every student, and to encourage more students to participate in and 
succeed in advanced mathematics and science courses. 

{Goal 2) To increase and sustain the number, quality, and diversity of preK-12 teachers 
of mathematics and science, especially in underserved areas, through further development 
of a professional education continuum.... 

{Goal 3) To contribute to the national capacity to engage in large-scale reform through 
participation in a network of researchers and practitioners that will share, study and 
evaluate educational reform and experimental approaches to the improvement of teacher 
preparation and professional development. 

{Goal 4) To engage the learning community in the knowledge base being developed in 
current and future NSF Centers for Learning and Teaching and Science of Learning 
Centers, (pp. 2-3) 
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MSP program theory of change. As mentioned, a program’s theory of change is more than 
identifying ends and means; it includes predictive assumptions about why taking a certain course 
of action will attain a desired outcome. In the case of the MSP program, our best inference about 
the primary theory of change in the 2002 solicitation appears to be this: 

If preK-12 educational systems and institutions of higher education can find ways to 
develop and sustain fruitful partnerships that address the program’s four major goals (i.e., 
increasing participation, transforming professional development, etc.), then participating 
schools will increase their capacity for meeting high standards for learning and for 
significantly reducing achievement gaps in the mathematics and science performance of 
diverse student populations. These approaches, effectively implemented, will foster 
“systemic” improvements in math and science education across the PK-12 and 
postsecondary systems. 

Although this theory of change makes clear the programmatic means (school-university 
partnerships) for pursuing its desired ends (wide-scale improvement of student achievement), it 
offers no explanation for why this approach will attain these desired improvements. That is, what 
sort of reasoning might explain why this approach would have more success than other 
approaches? What evidence can be lined up that would make these predictive assumptions 
credible and defensible? In the 2002 MSP solicitation, which not only attracted dozens of 
proposals but also was a basis for determining which proposals were funded, the rationale for 
using partnerships to improve student achievement is not obvious. Admittedly, NSF program 
solicitations traditionally do not expound at length on the research bases for designing a program 
in a particular way. Still, considering that the National Science Foundation had allocated $160 
million to the MSP program, it is somewhat surprising that the solicitation does not offer more to 
explain why this kind of program would achieve its desired outcomes. 

In our examination of the MSP program’s theory of change, we also looked for theories of 
change in three comprehensive projects funded through the first cycle of the program. What 
follows are summaries of each project and the theories of change we inferred from them. 

1. The El Paso Math and Science Partnership (award: approximately $29.5 million) 
includes three urban school districts that encompass El Paso, nine rural school districts, the 
University of Texas at El Paso (UTEP), El Paso Community College, the Region 19 Education 
Service Center, and El Paso area civic, business and community organizations and leaders. Like 
other MSPs, the El Paso MSP aims to improve student achievement in mathematics and science 
among all students, at all preK-12 levels, and to reduce the achievement gap among groups of 
students. The five main goals of this particular partnership include (1) engaging university and 
community-college leadership and mathematics, science, engineering and education faculty in 
working toward significantly improved K-12 math/science student achievement; (2) ensuring the 
number, quality, and diversity of K-12 teachers of mathematics and science across partner 
schools, particularly those with the greatest needs; (3) building the capacity of area districts and 
schools to provide the highest quality curriculum, instruction, and assessment, and to ensure the 
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highest level achievement in mathematics and science for every student; (4) ensuring a K-16 
alignment of mathematics and science curriculum, instruction and assessment; and (5) 
prioritizing research on educational reform and preK-16 partnerships. 

2. The Milwaukee Mathematics Partnership: Sharing in Leadership for Student 

Success (award: approximately $20 million) comprises the University of Wisconsin-Milwaukee, 
the Milwaukee Public Schools, and the Milwaukee Area Technical College. The Milwaukee 
Mathematics Partnership seeks to substantially improve mathematics achievement for the 
100,000 K-12 Milwaukee Public Schools students through achieving the four following goals: 

(1) to use a Comprehensive Mathematics framework to lead a collective vision of deep learning 
and high-quality teaching across the Partnership; (2) to institute a distributed mathematics 
leadership model based in professional learning communities; (3) to develop a Teaching 
Learning Continuum that builds and sustains the capacity of teachers to use a deep personal 
understanding of math to improve student achievement; and (4) to develop a Student Learning 
Continuum to ensure all pK-16 students have access to, are prepared and supported for, and will 
succeed in challenging mathematics. 

3. The Appalachian Math Science Partnership (award: approximately $25 million), which 
includes 38 Kentucky school districts, nine Tennessee school districts, five Virginia school 
districts, the Kentucky Science and Technology Corporation, and 10 institutions of higher 
education led by the University of Kentucky, seeks to strengthen and refonn education in math 
and science in pre-K through Grade 12 classrooms in participating districts mainly by building 
an integrated elementary, secondary, and higher education system in this underserved region. 

The partnership will unite the efforts of teachers, administrators, guidance counselors, and 
parents in local schools with administrators and faculty at area colleges and universities. 
Collaborations among these stakeholders from numerous regional education systems—both K-12 
and postsecondary—will meet felt needs for (1) preservice teacher and administrator education, 

(2) professional development of preK-12 personnel; (3) student learning opportunities, including 
parent/community enrichment; and (4) research that advances understanding of rural education 
reform; addressing these needs will lead to greater student achievement in math, science, and 
technology. 

Project theories of change. All three MSP projects described above have the same general 
theory of change: If critically important stakeholders inpK-12 and postsecondary systems 
collaborate to design and implement better ways to prepare and support pre-sendee and current 
teachers in math and/or science, then students in participating schools and districts will achieve 
better scores in math and science. Because the projects we sampled have similar goals and 
strategies that strongly reflect the MSP’s embedded theory of change, we believe they are largely 
implementing the program’s vision of how improvements in math and science education are to 
be achieved. The project summaries are nearly interchangeable, and the theories of change of the 
three projects and the MSP program are tightly aligned. However, as these are never overtly 
stated, they have to be inferred from description of goals and tactics. Rationales for why these 
strategies might achieve these goals are not offered. 
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II. Course, Curriculum, and Laboratory Improvement Program 

As expressed in its 2007 program solicitation, the primary goal of the CCLI program is to 
“stimulate, evaluate, and disseminate innovative and effective developments in undergraduate 
STEM education” (p. 4). To achieve this overarching goal, the CCLI program funds projects that 
it believes will introduce new content incorporating cutting-edge developments in STEM fields; 
produce knowledge about learning; and improve educational practice. The relationship between 
knowledge production and improvement of practice in undergraduate STEM education is 
represented by a cyclic model with five components. 



Figure 1. The CCLI Program Cyclic Model 


According to the CCLI solicitation, 

In this model, research findings about learning and teaching strategies that show promise 
give rise to faculty development programs and methods that incorporate these materials. 
The most promising of these developments are first tested in limited environments and 
then implemented and adapted in diverse curricula and educational institutions. These 
innovations are carefully evaluated by assessing their impact on teaching and learning. In 
turn, these implementations and assessments generate new insights and research 
questions, initiating a new cycle of innovations, (p. 5) 

The CCLI program solicits proposals for three types of projects that represent three phases of 
development. Phase 1 projects typically address one program component and involve a limited 
number of students and faculty at one academic institution; their maximum project budget is 
$150,00 for 1-3 years. Phase 2 projects build on smaller-scale successful innovations or 
implementations to refine and test these on diverse users in several settings; the maximum 
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budget is $500,000 for 2-4 years. Phase 3 projects combine established results and mature 
products from several components of the cyclic model, and, drawing upon a diversity of 
academic institutions and student populations, strive to achieve a demonstrable national impact; 
maximum budget for Phase 3 proposals is $2,000,000 over 3-5 years. Finally, regardless of 
project scope and budget, all proposed projects were asked to incorporate what the solicitation 
described as “important program features”: a focus on students; use of and contribution to 
knowledge about STEM education; STEM education community-building; expected measurable 
outcomes; and project evaluation. 

CCLI theory of change. Unlike the newer MSP program, which we claim does not have a 
well-articulated theory of change, the more-established CCLI program is based on a theoretical 
cyclical model that assumes several things. First, it appears to assume that efforts to improve 
undergraduate STEM education should start small with “grass roots” efforts at change. This is 
demonstrated by two features of the program solicitation: the option of proposing small-scale 
projects willing to experiment on a product or activity that falls within one of the five 
components of the cycle; and the requirement that larger projects must be scaled from successful 
trials in smaller projects. Thus, implicit in the CCLI theory of change are assumptions about the 
scalability of innovations—and other research suggests these assumptions about scalability may 
be unwarranted. As Seymour (2007) points out, despite an investment of significant resources— 
namely, large amounts of money by the NSF and other funders to seed innovation, the 
establishment of numerous networks of reform-oriented faculty, and the accumulation of 
scientific and practical knowledge about how students learn—the spread of research-grounded 
teaching practices in U.S. undergraduate education remains limited. In asserting that STEM 
education reform has stalled, DeHaan (2005) points to the finding of a 2001 survey of research¬ 
intensive universities: Only small numbers of students at approximately 20% of these institutions 
have opportunities to take introductory courses that feature active learning or real-world problem 
solving. Thus, it may be riskier than the NSF believes to assume that large-scale reform in STEM 
education will come from encouraging the scaling up of successful small projects. 

Second, the cyclic model assumes that its five components are in sequence. As the 
solicitation explains, creating innovative learning materials and teaching strategies should be 
“guided by research on teaching and learning, by evaluations of previous efforts, and by 
advances within the disciplines” (p. 6). In turn, new materials will lead first to develop faculty 
expertise and then implement the innovation in actual educational settings, the success of which 
at improving student learning must be assessed. Although the model stipulates that any of the 
five components can be a starting point in the cycle, it li nk s the components with one-way 
arrows, specifying in the solicitation text what must precede and follow each. This feature of the 
model reflects unstated assumptions about how innovations are scaled up, one assumption being 
that achieving large-scale improvements in undergraduate STEM education must involve all five 
components. Moreover, it attempts to link an organic, “bottom-up” approach to change that 
drives the smaller projects with one that is sequential and (absent any proof of concept) unproven 
as to whether the cycle truly functions as expected. We also do not leam whether and how 
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discoveries made in one funded project are disseminated to potential users and their adoption 
realized. There seem to be no extra-project, intra-program mechanisms to make the model work. 

In short, although the cyclic model brings greater conceptual clarity to how this particular 
NSF program will achieve its primary outcomes, the solicitation does not necessarily articulate 
why these means are equal to or better than other approaches to attaining the same aim. Although 
the CCLI program model does a better job than the MSP program of specifying a particular 
approach to change, it really does not explain in convincing terms why the cyclical approach is 
more likely to succeed than another approach. 

In examining the theories of change in projects funded by the CCLI program, we focused on 
three Phase I projects. What follows are summaries of each project and the theories of change we 
inferred from them. 

1. Authentic, Career-based, Discovery Learning Projects in Introductory Statistics 

(award: $176,000) aims to improve achievement and change attitudes for students who take a 
first statistics course in either high school or college by developing and testing a model for 
learning in statistics that suggests that career-specific discovery learning will enhance outcomes 
for students. Supplementary materials include a faculty guidebook to help design discovery- 
learning projects and a workbook of introductory level problems from specific majors. The 
initiative fosters collaboration between mathematics and other departments, especially those in 
the social and behavioral sciences, business, education, and health sciences. 

2. Developing Engineering Lifelong Learners through Freshman Seminars and Faculty 
Development Workshops (award: $150,000) is developing a seminar course for first-year 
engineering students that will instill in them a commitment to lifelong learning and associated 
skills such as self-directedness, metacognition, and critical reflection. The courses are organized 
around a portfolio project through which students develop a vision of engineering as a profession 
and make a conceptual connection between mathematics and science and this vision. Faculty 
development workshops will assist faculty in delivering the seminar course. 

3. Building a Basic Biology Concept Inventory (award: $285,000) is developing a basic 
Biology Concept Inventory that will enable the field to reliably quantify student learning at the 
introductory college level. The objective is to provide the field with reliable data on student 
learning as biology departments around the country attempt to improve student achievement. The 
existence of a Biology Concept Inventory is argued to have the potential to impact the teaching 
of biology to thousands of undergraduates throughout the country, much as the Force Concept 
Inventory and the Astronomy Diagnostic Test have influenced the teaching of Physics and 
Astronomy. 

Project theories of change. Perhaps in keeping with the more organic type of change 
encouraged in Phase I-type projects, the theories of change we have inferred from these three 
projects are all quite different. The first project, for example, is betting that situating statistics in 
career-related topics and problems is an approach that will improve the achievement and 
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attitudes of students in their first statistics course. The second project believes that a seminar for 
first-year engineering students is a good way to cultivate a disposition in engineers toward 
lifelong learning. The third project is the most research-based, assuming that if it can create and 
test a biology concept inventory, then it will be adopted by instructors who want to identify and 
rectify pervasive misconceptions in biology courses. Their diversity notwithstanding, these 
projects incorporate key features of the CCLI program’s theory of change (e.g., a focus on 
students, contributing to STEM education knowledge base, etc.). This short discussion precludes 
a close examination of the particular theories of change in these three projects (e.g.. Is a single 
seminar enough to make lifelong learners? If you build a biology concept inventory, will the 
faculty use it?). However, we still can say that none of the three projects appears to put forth the 
kind of theory of change that explains not only why its particular approach to solving its key 
problem will work but also how the good outcomes it achieved will be taken up and used more 
widely, how changes will be sustained, and how the products of several successful trials can be 
combined and institutionalized in departments, across disciplines, or spread throughout the 
education system. 

TIT: Systemic Changes in Undergraduate Chemistry Curriculum 

Between 1995 and 2002, the Systemic Changes in the Undergraduate Chemistry Curriculum 
program sought to bring about major improvements in the undergraduate chemistry curriculum 
and instruction by building on the most promising results of previous NSF-funded work. This 
program is distinctive in that it chose large, multi-institutional projects that built on smaller 
grants made over the previous decade through other NSF programs. In the following account, 
information contained in formal program and project descriptions is augmented by information 
provided by project evaluators. 

Program theory of change. Our inferences about the program’s theory of change stem from 
the nature of the projects chosen for funding. The essential strategy of the Systemic Changes in 
Chemistry Curriculum program was to “pick winners”; it selected five projects that had already 
begun to address the key problem of how to encourage the spread of research-grounded methods 
of teaching and learning, and curricula that addressed real world issues. These were cross¬ 
institution collaborations that had already been effective on a smaller scale under these programs. 
They became, essentially, larger-scale experiments in how to disseminate promising practices. 

This program was the first major initiative to promote real-world relevance in science 
curricula, and all of the funded projects developed and promoted teaching materials that reflected 
this focus. Thus, one (of several) theories implicit in this program was that students will become 
interested and engaged in chemistry, and will understand it more deeply, if they can see the 
significance of what they are learning for their own lives and for the world around them. 

Embedded in project selection was also the theory of dissemination by peer-led professional 
development. All selected projects used peer-to-peer workshops in which faculty (and student 
teaching assistants) taught each other how to use research-grounded teaching methods, and drew 
in increasing numbers of chemistry faculty who were interested in (or already experimenting 
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with) these methods. Thus, these projects were a testing ground for the theory that widening 
uptake of best practices can be leveraged by hands-on encoun ters with new teaching methods in 
a risk-free, supportive, and highly convivial network of peers. Contacts made in workshops 
continued in meetings in which class and lab materials were developed, and commitment to 
“reformed” teaching methods was reinforced. The (implicit) theory of change in this method was 
effective dissemination by infectious engagement. 

The program was also, in effect, funding grassroots activity in which individual and small 
groups of faculty were engaged with faculty from other chemistry departments. The learning 
communities built up in four of the five projects were cross-institution rather than within- 
institution networks. Engagements by whole departments were rare, and participating (notably 
pre-tenured) faculty were somewhat sheltered from the disapproval of departmental colleagues 
by support from senior members of the project network. 

Regional workshops that carried the growing body of faculty teaching expertise, methods, 
and materials to a widening circle were funded for several years after the 5-year project funding 
had ended. A sister program gave awards to department-based groups that undertook to “adapt 
and adopt” the methods and materials of the core project groups. Although (as reported by 
project evaluators) both the project participants and the program officers implicitly understood 
the methods they were developing—often as they went along—and the rationale that supported 
them, these were not explicitly stated. 

It is clear that the program was “systemic” in intent because it encouraged applicants to 
spread new methods and materials to colleagues in science teaching preparation, community 
college science faculty, high school science teachers, and college teaching assistants. Although 
project proposals expressed an intention to address these ambitious goals, the grassroots 
organizations that ran the projects were limited by organizational constraints and thus varied in 
the extent to which they could meet them. 

Below we describe three of the five programs funded under this program. The first two 
projects, ChemLinks Coalition and Modular Approach to Chemistry Curriculum Reform, were 
persuaded by program officers to work as one cooperating project—“ChemConnections”— 
because of the similarity of the two projects’ goals and methods. 

1. ChemLinks Coalition: Making Chemical Connections (award: $2.8 million) was a 5- 
year collaboration with the ModularChem Consortium to change the way students learn 
chemistry, increase scientific literacy for all students taking chemistry, and promote a process of 
educational reform. Participating faculty developed, tested, and disseminated modular course 
materials that use active and collaborative approaches to learning. These materials, targeted to 
the first 2 years of the chemistry curriculum, drew upon real-world questions important to 
students and society that could be answered only by learning and applying the necessary 
chemistry. Questions often addressed environmental issues from the perspective of personal 
choices, such as “Would you like fries with that?” “How do I know what’s in my drinking 
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water?” and “Paper or plastic?” The project focused largely (though not exclusively) on 
introductory chemistry courses that included non-science majors and those taking chemistry as a 
supporting course, as well as chemistry majors. It sought to interest and engage students who are 
fearful of taking chemistry and find it difficult. The project was undertaken by a coalition of 
leading liberal arts colleges and research universities that already had experience working 
together on chemistry curricular reform, and leveraged Project Kaleidoscope’s extensive network 
to involve a larger and more diverse group of institutions in making systemic and sustainable 
changes in undergraduate chemistry education. 

2. Sweeping Change in Manageable Units: A Modular Approach for Chemistry 

Curriculum Reform (award: $3 million) sought to develop new curricula, materials and 
methods to encourage the appreciation and learning of science, especially chemistry, by every 
undergraduate student. In collaboration with the ChemLinks Coalition, this project developed, 
tested and refined modules at the 2- and 4-year colleges and research universities comprising the 
two consortia. Modules lasting 1-4 weeks were designed that presented fundamental chemistry 
to students in the context of a real-world problem or application and emphasized the li nk s 
between chemistry and other disciplines. Like ChemLinks, ModularChem emphasized teaching 
methods that reflected current research in the learning sciences and encouraged active learning. 
The project constructed a framework for the continuous improvement of curricula, and shared 
workshops at national and regional meetings with ChemLinks and members of the other 
chemistry project to disseminate their work. 

3. Workshop Chemistry Curriculum (award: $1.6 million), a consortium of 10 senior and 
community colleges at the City University of New York, and the Universities of Pittsburgh, 
Pennsylvania, and Rochester, developed and disseminated a new model of teaching, Workshop 
Chemistry, which provided a collaborative learning experience that increased student 
involvement and involved students as mentors. A prototype workshop model developed at City 
College in a general chemistry course for science and engineering majors was expanded and 
refined for a broad range of courses including preparatory chemistry, chemistry for allied health 
sciences, organic chemistry, instrumental, and analytical chemistry. Students who served as 
workshop leaders were given a natural introduction to teaching that was formalized through a 
teacher preparation component of the project. Student workshop manuals that include the 
problem solving, model building, and simulation activities of the workshops were produced for 
each course and disseminated. New project partners were invited to view workshops, participate 
in faculty development activities, and implement pilot workshop courses at their own 
institutions. 

Project theories of change. Looking across these three programs, we note the themes of 
making chemistry accessible to a wider student range through real-world relevant science, and 
the dominant strategy of dissemination via expansion of cross-institutional faculty networks as 
learning communities. The embedded theories of change that we have described were largely 
developed in practice by the project participants themselves with reference to a broad outline 
offered by the program. 
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The two closely related projects of the ChemConnections collaboration produced modular 
teaching materials that reflected the theory that students will become interested and engaged in 
chemistry when they see its relevance. Thus, real-world content makes chemistry more 
accessible. As the modules incorporated active, interactive, and discovery-based pedagogies to 
convey the reconceptualized content (and, in some cases, learning assessment methods), they 
reflected the theory that students learn more deeply when what they learn and the ways in which 
they learn it are coherently related. This project was also committed to multi-media teaching 
methods (which were incorporated into many of the modules), which implies a theory that 
teaching with technology promotes effective learning. 

Although this project’s leaders originally envisioned a conventional dissemination strategy 
based on presentations at disciplinary meetings, they quickly joined in the process of using 
regional peer-led workshops, work-focused (i.e., module development) meetings, and learning 
community building methods of their ChemLinks partner. These occasions became the meeting 
place for collaborators from community colleges, historically black colleges and universities, and 
high schools. Thus, by experiment rather than prior design, the twin projects embraced the theory 
that reform initiatives can be effectively disseminated and articulated across different 
educational levels by regular meetings of network members in which real work (such as 
curriculum and teaching methods development) is done in an egalitarian, collegial atmosphere. 
In their reports, the projects’ evaluators noted the high level of involvement of community 
college chemistry faculty in network events. 

The third chemistry curriculum initiative, Workshop Chemistry, enacted a theory of change 
that, though similar to the two other projects, differed with respect to the role that students could 
take in learning chemistry. Workshop Chemistry’s team-learning model of instruction meant that 
students who did well were given an opportunity to become peer leaders who coached their 
classmates and worked closely with course instructors. Thus, central to this project’s theory of 
change is the assumption that students learn chemistry better when they (a) take a structured role 
in helping their peers learn the subject matter and (b) can appren tice with faculty to learn how 
to foster greater student learning. 


Summary of Findings 

In the case of the MSP program, its theory of change consisted of predictive assumptions 
about the effects that partnering preK-12 programs with institutions of higher education would 
have on student achievement in math and science. As we noted, these assumptions were not 
directly articulated in the program solicitation but were treated as a given. The theories of change 
of the projects receiving awards hewed closely to the MSP programmatic theory of change. 

Thus, this implicit theory of change about the impact of partnerships was accepted by and 
reflected in the projects. Perhaps the enormous stakes attached to getting a multimillion dollar 
NSF award contributed to uncritical adoption of the program’s predictive assumptions about 
preK12- postsecondary partnerships. The alignment between projects and the program 
solicitation suggest an implementation approach to changing STEM education—that is, the 
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projects collectively bought into the program’s predictive assumptions and are carrying out its 
change strategy. 

Compared with the MSP program solicitation, the programmatic theory of change in the 
CCLI solicitation was more evident, explicitly drawing upon a formal model that assumes that 
scaling up STEM refonn is achieved through a cycle of curriculum development, faculty 
development, implementation, assessment, and research. However, the solicitation did not 
articulate by what mechanisms successful projects would leverage uptake of their methods and 
materials, that is, how this model could actually work in scaling up the results of successful 
experiments. It also includes no explanation of why this model is fit for achieving its purpose. As 
with the MSP program solicitation, the wager the NSF placed on this approach is treated as an 
obvious bet. Despite its formal design, this is essentially a hands-off program that is closest to 
competitive research. The three CCLI Phase I projects we examined employed different theories 
of change; this diversity seems characteristic of Phase I awards. In contrast to an implementation 
program like the MSP partnership, the CCLI program is more of an experimentation program. 

Finally, in the case of the Systemic Changes in the Undergraduate Chemistry Curriculum 
program, its theory of change, like that of the MSP program, was largely unstated. However, 
unlike the MSP program, which was more prescriptive, the chemistry curriculum program had a 
strategy of looking to previously funded projects that had successfully used variations of peer-to- 
peer professional development to improve how undergraduate chemistry was organized and 
taught. 

The chemistry curriculum program is one of NSF’s first major efforts to leverage 
improvements across and beyond the STEM higher education system. It promotes several strong 
objectives, all of which are represented in the five projects it funded. It charges these projects to 
be system change experimenters, which are to figure out in practice how broadly agreed 
principles of dissemination can be made to work. Both the projects’ successes and barriers 
encountered make them the active authors of an evolving theory of system change. These 
projects offer good proof of concept for the peer-led model of professional development and 
diffusion of improved pedagogies, real-world relevant curricula, and learning assessment 
methods that express learning objectives: many of their products and methods are still in wide 
use. How far this theory of change, as developed and refined by this set of projects, might have 
gone in disseminating these practices and building faculty capacity to sustain them is unknown 
because the program’s theory of change did not account for the time it would take to establish 
new methods. Funding ceased after 5 years, and workshop dissemination funding was 
discontinued notwithstanding strenuous efforts by the five collaborating projects to sustain it. 
This raises an important question about whether the NSF should proactively sustain and extend 
experiments in change that show good results rather than following a research model that funds 
largely innovative experiments like the CCLI program. 
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Implications 

As we noted, Weiss (1995) argues that programs trying to bring about change are predicated 
on assumptions about the relationship between actions and outcomes. Theories of change reflect 
important yet often invisible choices by program designers to select one change strategy over 
others. Making any kind of guess—educated or otherwise—about the best course to achieving a 
desired but uncertain end is always a wager of sorts, and, as such, a theory of change is a kind of 
bet that one’s approach to change is, in light of the situation at hand, the best course to take. It 
involves appraising not only the potential benefits of the program but also all the factors that 
affect the likelihood that the chosen course of action will be better than its alternatives. Although 
these choices and predictive assumptions are important considerations in program design, they 
are, Weiss asserts, often hidden from plain sight and thus cannot be evaluated as to their 
soundness. Over time, an implicit theory of change may come to be taken as the status quo and 
no longer seen as a bet made by program designers in the past. 

What our study of NSF programs and projects shows about Weiss’s claim is that program 
solicitations do not typically disclose the full chain of primary assumptions and key choices that 
program officers make in designing a program; if they did (as Weiss demonstrates in her own 
example of the job training program), then project proposers, principal investigators, and other 
project designers could themselves critically consider which assumptions would and would not 
hold in their local contexts. Moreover, the solicitations we examined suggest that implicit 
theories of change still profoundly shape the theories of change in projects that respond to them. 
A danger of designing funding programs with unarticulated theories of change, then, is that the 
projects they attract and support may not recognize the assumptions on which they are organized; 
this lack of insight may affect a project’s ability to achieve the program’s desired ends. Without 
a clear idea of why programs work as they do and what makes programs more or less successful, 
then NSF programs and projects must keep rediscovering what brings about desired change. 
Based on our examination of a few programs and projects, we suggest that NSF has a blind spot 
with respect to seeing the need for making its theories of change explicit. This blind spot may be 
one reason why key aims of the NSF’s Division of Undergraduate Education—namely, 
sustainability and expansion of practices proven to encourage greater student learning in math 
and science—have persistently eluded the NSF. 

Any theory of change—whether new or synthetic—that hopes to effectively mobilize STEM 
education for a sustainable future will need to avoid the pitfalls this paper has discussed, and be 
guided by the wisdom that Weiss and her colleagues offer. Developing clear and coherent 
theories of change that are appropriate to the scale of the endeavor, and are as comprehensive as 
the scope of the vision may demand, will require that project planners realistically appraise the 
structural and cultural factors that can enhance or limit chances of success. They will need to 
explain why the approaches suggested are likely to work, and how the elements in the resulting 
proposed design can be articulated. In struggling with this fundamental challenge, those 
developing theories of change will contribute to a better understanding among policymakers, 
funders, and education reformers how to use theories of change to best advantage. 
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